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-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

Responsive to communication(s) filed on 29 May 2003 . 
2a)£3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 1-13 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E3 Claim(s) 1-13 is/are rejected. 

7) Q Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

1 1) £3 The proposed drawing correction filed on 07 April 2003 is: a)D approved b)^ disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

1 3) E3 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 

a)IElAII b)D Some*c)D None of: 

1 .£3 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1) ^ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 



2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) Q Notice of Informal Patent Application (PTO-152) 

3) £3 Information Disclosure Statement(s) (PTO-1449) Paper No(s) 12 . 6) ^ Other: Translation ofJP 07-319525 . 
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DETAILED ACTION 
Drawings 

1. The proposed drawing correction filed on April 7, 2003 has been disapproved because it 
is not in the form of a pen-and-ink sketch showing changes in red ink or with the changes 
otherwise highlighted. In addition, it would be preferable if each axis of the coordinate axes in 
Figures 1-3 were labeled for example as x-, y-, and z-axes. See MPEP § 608.02(v). 

Double Patenting 

2. Applicant is advised that should claim 9 be found allowable, claim 12 will be objected to 
under 37 CFR 1.75 as being a substantial duplicate thereof. When two claims in an application 
are duplicates or else are so close in content that they both cover the same thing, despite a slight 
difference in wording ("orientation" in claim 9 compared to "arrangement" in claim 12), it is 
proper after allowing one claim to object to the other as being a substantial duplicate of the 
allowed claim. See MPEP § 706.03(k). 

Claim Objections 

3. Claims 1 and 2 are objected to because of the following informalities: the amendment 
filed April 7, 2003 amended claims 1 and 2 such that after the word "and" in line 3 of both 
claims appears an apparently erroneous right bracket ("]"). Appropriate correction is required. 

Response to Arguments 

4. Applicant's arguments, see page 6, middle of the page, filed April 7, 2003, with respect 
to a prior art label for Figures 11-13 have been fully considered and are persuasive. The 
objection of Figures 1 1-13 has been withdrawn. 
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5. Applicant's arguments, see page 7, second to last paragraph, filed April 7, 2003, with 
respect to the 35 USC 112, 2 nd paragraph rejection of claims 1-3 have been fully considered and 
are persuasive. The rejection of claims 1-3 has been withdrawn. 

6. Applicant's amendment has required new grounds of rejection, which are set forth in the 
following Office action. Applicant's arguments with respect to claims 1 and 2 have been 
considered but are moot in view of the new ground(s) of rejection. 

7. Regarding applicant's arguments on page 8, second to last paragraph, applicant states that 
"Suzuki mentions the robot using a camera during construction, but it does not mention any 
detail of how that camera is used." The examiner respectfully disagrees. Suzuki discloses that 
2-dimensional images of sample 10 are acquired by moving CCD camera pair 11a and lib 
around the sample, or by rotating the sample, in order to obtain images from various directions to 
determine 3-dimensional positions of the vertices and edges of the sample. Suzuki further 
discloses a construction recognizing circuit that acquires part arrangement data including 3- 
dimensional positions and orientations of the parts included in the sample, which is stored in 
memory. Part detecting circuit (18) compares the layer outline data of each height obtained by 
position sorting circuit (17) with form data of each part stored in part-form memory (19), thereby 
specifying parts included in each layer (column 3, line 66-column 4, line 24). 

Claim Rejections - 35 USC § 112 

8. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

9. Claims 5-7 are rejected under 35 U.S.C. 112, second paragraph, as being indefinite for 
failing to particularly point out and distinctly claim the subject matter which applicant regards as 
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the invention. Claims 5-7 (depending from claim 2) recite the limitation "said image pickup 
device" in lines 2 and/or 3. There is insufficient antecedent basis for this limitation in the claims. 
It appears that applicant intended to change line 12 of claim 2 from "an image pickup means" to 
"an image pickup device" as in amended claim 1 . 

Claim Rejections - 35 USC §103 

10. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

11. Claims 1-6 and 8 are rejected under 35 U.S.C. 103(a) as being unpatentable over Suzuki 
(USPN 4,835,450, reference of record) in view of Werth et al. (USPN 4,504,970, reference of 
record), hereafter Werth. 

12. Regarding claims 1 and 8, Suzuki discloses a teaching model generating method and 
device for image processing, in which a subject object has the same or substantially similar shape 
as that of a reference object (column 3, line 66 to column 4, line 9), and the device comprising: 
an image processing system with which current three-dimensional position and/or posture of the 
subject object is recognized (determined) based on a plurality of predetermined teaching models 
of the reference object (column 4, line 10-24); and an image-capture system, in advance of the 
recognizing (determining), generating and storing the plurality of teaching models on the basis of 
respective image data produced by taking images of said reference object from a plurality of 
directions (column 4, line 25-39), wherein one of the reference object and an image pickup 
device is fixed to a movable part of a robot or is grasped with a hand of the robot (either moving 
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the pair of CCD cameras 1 la and 1 lb around fixed sample 10, or rotating sample 10 in front of 
the fixed cameras), and said robot is operated for positioning to a plurality of different image 
pickup positions and directions (column 3, line 66 to column 4, line 9), so that the image data 
respectively obtained at each of said different image pickup positions is stored as a teaching 
model (column 4, line 32-39). 

13. Suzuki does not explicitly disclose carrying out pattern matching processing of an image 
of the subject based on the plurality of predetermined teaching models of the reference object. 
Werth discloses a training controller for a pattern processing system applicable to a wide range 
of different pattern processing requirements including visual image recognition (column 3, line 
37-44). Werth discloses that an operator trains the system to recognize inspected parts by 
assigning a training code to a correctly constructed part (teaching model) such as a printed circuit 
board in its registered position (column 5, line 26-column 6, line 6). It would have been obvious 
to one of ordinary skill in the art at the time the invention was made to carry out pattern 
matching processing of the subject image as taught by Werth in order to provide improved 
pattern recognition with a dramatic reduction in data by identifying image input patterns based 
upon address loops to associate a desired response with an image input, thereby providing a 
much faster response to complex input images (column 15, line 35-43). 

14. Regarding claim 2, Suzuki discloses a teaching model generating device for image 
processing, in which a subject object has same or substantially similar shape as that of a 
reference object, and the device comprising: an image processing system with which a current 
three-dimensional position and/or posture of the subject object is recognized by carrying out 
pattern matching processing of an image of the subject based on a plurality of pre-determined 
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teaching models of the reference object; and an image capture system, in advance of the 
recognizing, generating and storing the plurality of teaching models on the basis of respective 
image data produced by taking images of said reference object from a plurality of directions, 
wherein a robot is operated for positioning to a plurality of different relative image pickup 
positions and directions, so that the image data respectively obtained at each of said different 
image pickup positions is stored as a teaching model (see discussion of claim 1 above). 

15. Suzuki does not, however, disclose that the reference object is fixed to a movable part of 
a first robot or is grasped with a hand of the first robot, and an image pickup means is fixed to a 
movable part of a second robot or is grasped with a hand of the second robot. Werth discloses a 
training controller for pattern processing system wherein it is suggested that an application could 
utilize two robot arms, one which holds a camera which visually guides it to observe a precise 
assembly point and a second which brings a tool or assembly within the visual field of the 
camera where it is visually guided through an operation (column 5, line 12-17). It would have 
been obvious to one of ordinary skill in the art at the time the invention was made to utilize two 
robot arms, one to hold a camera and one to hold a tool or assembly as taught by Werth in order 
to provide more degrees of freedom allowing more views of the workpiece from different 
directions and to provide proper alignment for mating parts in automated assembly operations or 
move a tool to a specific point on the part (column 5, line 6-11). 

16. Regarding claim 3, Suzuki discloses that said teaching model is a part of the image data 
of the reference object (column 4, line 32-39). 
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17. Regarding claim 4, Suzuki discloses that said teaching model is composed of data 
obtained by performing image processing (3-D position detecting circuit 16) on the image data of 
the reference object (column 4 5 line 5-9). 

18. Regarding claim 5, Suzuki discloses that said teaching model is generated for every 
direction in which said image pickup device (means) took the image of said reference object 
(column 4, line 2-9) and said teaching model is stored in association with the information on the 
direction (column 4, line 32-39). 

19. Regarding claim 6, Suzuki discloses that said image pickup device (means) is a camera 
(column 3, line 30-34). 

20. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Suzuki in view of 
Werth as applied to claim 2 above, and further in view of Ninomiya et al. (USPN 4,61 1 ,292, 
reference of record), hereafter Ninomiya. 

2 1 . Suzuki does not disclose that said image pickup means (device) is a three-dimensional 
visual sensor whose image pickup means measures the distance between the image pickup means 
and a plurality of points on the object. Ninomiya discloses a robot vision system including a 
three-dimensional visual sensor whose image pickup means measures the distance between the 
image pickup means and a plurality of points on the object (column 4, line 28-49). It would have 
been obvious to one of ordinary skill in the art at the time the invention was made to utilize a 
three-dimensional visual sensor as taught by Ninomiya in order to determine the position and 
posture of an object without operation or accuracy being effected by contrast, color, or surface 
condition of the object (column 10, line 35-39). 
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22. Claims 9-13 are rejected under 35 U.S.C. 103(a) as being unpatentable over Suzuki in 
view of Werth as applied to claims 1 and 2 above and further in view of Iida et al. (JP 07- 
319525, reference of record, English translation included herewith), hereafter Iida. 

23. Regarding claims 9 and 12, Suzuki discloses a method of automatic orientation 
recognition, comprising: generating and storing a set of images of different relative orientations 
(arrangements) of a subject, the images having been captured by a plurality of robotic operations 
corresponding to the different relative orientations (arrangements) of the subject, and associating 
with each image information indicating its respective relative orientation (arrangement) of the 
subject (column 3, line 66-column 4, line 24); with a known current orientation (arrangement) of 
a robot, capturing a current image of a workpiece, where the workpiece has a shape substantially 
similar to the shape of the subject (column 4, line 49-column 5, line 1 1); Werth discloses using 
pattern matching to match one of the stored images with the current image (column 5, line 26- 
column 6, line 6; see above discussion of claims 1 and 2); and Suzuki discloses determining the 
orientation (current arrangement) of the workpiece relative to the robot based on the relative 
orientation (arrangement) information associated with the matched stored image (column 4, line 
25-39), and also based on the known current orientation of the robot (column 3, line 50-62). 

24. Suzuki discloses that the positions where the robot grips the parts from the conveyor are 
automatically detected using cameras 11a and 1 lb (column 4, line 49-68), but does not explicitly 
disclose that the captured image includes a workpiece having an unknown orientation relative to 
the robot. Iida discloses a high-speed picking device for piled parts wherein an image of piled 
workpieces is captured in order to recognize the contour or border of a workpiece (W) in the pile 
using collation with two or more stored models including information on two or more robot 
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hands which grasp a specific part (page 2, paragraph 9-12). It would have been obvious to one 
of ordinary skill in the art at the time the invention was made to recognize a workpiece having an 
unknown orientation relative to the robot as taught by Iida in order to avoid having to pre-orient 
all parts before the robot can pick them up for assembly, inspection, or the like (page 1 , 
paragraph 3-4). 

25. Regarding claim 10, Suzuki discloses automatically maneuvering the robot to the 
workpiece based at least on the determined orientation of the workpiece relative to the robot 
(column 4, line 49-68). 

26. Regarding claim 1 1 5 Iida discloses that the generating and storing is performed for a 
plurality of differently shaped subjects (page 4, paragraph 32), wherein the current image 
includes a plurality of differently shaped (one of two or more specific) workpieces, and wherein 
the pattern matching (collating) further comprises identifying workpieces from among the 
plurality of differently shaped workpieces (page 2, paragraph 10) using the images and 
orientation (posture) information of the plurality of differently shaped subjects (page 2, 
paragraph 12-15). 

27. Regarding claim 13, Suzuki discloses a method comprising: robotically taking images of 
a subject with different subject-camera arrangements, and associating with each image 
information indicating its subject-camera arrangement (see above discussion of claims 1, 2, 8, 9, 
and 12). Iida discloses taking a current image of a workpiece shaped like the subject; and then 
determining a current workpiece-camera arrangement (posture) by matching (collating) one of 
the images (models) with the current image, and using predetermined subject-camera 
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arrangement information of the matched image to determine the arrangement of the workpiece 
relative to the camera (see above discussion of claims 9 and 12). 

Conclusion 

28. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. USPN 4,876,728 to Roth discloses a vision system for distinguishing touching parts. 
USPN 4,845,765 to Juvin et al. discloses a process for the automatic recognition of objects liable 
to overlap. USPN 4,985,846 to Fallon discloses an acoustical/optical bin picking system. USPN 
6,328,523 to Watanabe et al. (common inventor, commonly owned) discloses an apparatus for 
picking up articles using a vision-controlled robot. USPN 3,804,270 to Michaud et al. discloses 
a bulk storage and automatic presenter system with image producing means. 

29. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8 AM - 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone numbers for the 
organization where this application or proceeding is assigned are 703-872-9314 for regular 
communications and 703-872-9314 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-306-0377. 
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July 8, 2003 




